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ABSTRACT 


Tracking of moving objects in a sequence of images is one of the important 
and functional branches of machine vision technology. Detection and tracking 
of a flying object with unknown features are important issues in detecting 
and tracking objects. This paper consists of two basic parts. The first part 
involves tracking multiple flying objects. At first, flying objects are detected 


and tracked, using the particle filter algorithm. The second part is to classify 
tracked objects (military or nonmilitary), based on four criteria; Size (center 
Keywords: of mass) of objects, object speed vector, the direction of motion of objects, 
and thermal imagery identifies the type of tracked flying objects. 


oe To demonstrate the efficiency and the strength of the algorithm and the above system, 
JUE object , several scenarios in different videos have been investigated that include 
Flying object tracking challenges such as the number of objects (aircraft), different paths, 
Object detection the diverse directions of motion, different speeds and various objects. 
Particle filter One of the most important challenges is the speed of processing and the angle 
of imaging. 
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1. INTRODUCTION 

In addition to fixed objects, the real world often contains moving one. Hence, natural visual systems 
sensed and processed a continuous stream of observed scene data, and are less likely to encounter 
the perception of quite robust images. There is a similar situation in the machine vision. The visual system 
is often faced with a changing world that needs to be perceived and absorbed by successive images. 
Among the issues of interest in this regard is the question of tracking the motion of objects in a sequence of 
images. In this case, the visual system tries to detect the motion of objects on the scene by detecting 
sequential scenes from the scene during the period and identifying their movement paths. Object tracking 
is a significant field in computer vision [1]. The development of powerful computers, the availability of 
quality and cheap cameras, and the growing need for automotive video analytics has made a lot of attention 
to tracking algorithms for objects. Nowadays, visual aircraft detection is one of the interesting subjects in 
image processing projects because of the wide usage of low distance and low altitude aircraft. 

Several object tracking techniques have been suggested [2-6] in the video surveillance context. 
The oldest and most well-known is the Kalman filter. More latterly, several increasingly advanced methods 
were used - including Particle filters[7-11] Zhou et al. [12] uses the integration of color feature and contour 
knowledge in the particle filter based multi-hypothesis tracking algorithm. To enhance tracking performance 
unscented particle filter is a perfect choice to create relative distribution [13]. Iterative algorithms such as 
Particle swarm optimization (PSO) [14] are suitable to be implemented in tracking problems. Indoor tracking 
was implemented based on radio signal strength indication [15]. The paper aims are to provide an algorithm 
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for tracking flying objects in video images and then classify the type of tracked objects. In other words, 
identifying that classified objects are military or nonmilitary aircraft?. The paper is organized as follows. 
Detection and tracking of flying objects are presented in section 2. Section 3 is about the proposed 
classification type of flying objects (military or nonmilitary). The system simulation tool is introduced in 
section 4. We present experimental results in section 5 and the conclusion in section 6. 


2. FIRST PART: DETECTION AND TRACKING OF FLYING OBJECT 

Video analysis is the main task of a computer vision system that is used in many fields, including 
video surveillance, traffic monitoring systems, communication between humans and computers, maritime 
navigation, and so on. The analysis of video images consists of three basic steps: 

— Detect the moving object. 

— Follow the path of this object in sequential frames. 

— An object motion path analysis to identify its behavior, object type detection, and the issuance of 
required control commands. 

In the process of detecting objects by computer, interpretive elements are transformed into 
mathematical relations and implemented in the form of specific algorithms on the images. The important 
point in this discussion is the selection of appropriate features or parameters for extracting target information 
since the rest of the algorithm will create and overshadow this path. The question that is posed here 
is whether they are suitable features or how to choose a suitable set of features among the available features. 
It is very important to note that the function of detecting the system depends on the way objects 
are displayed. If the display of objects in a system is more accurate and based on more detailed information 
and features, then, that system will have more diagnostic power. Therefore, there is a determinant factor in 
choosing the power of display or resolution of existing data. On this basis, what is the process of discovery of 
the object and, consequently, the camouflage evaluation, operates so far as to be very dependent 
on this subject [16]. 

One of the first steps in processing an image is the image segmentation, which is one of the most 
difficult and most important stages in image processing. The segmentation of image processing has been 
considered widely in most branches of science and industry, and in many of these branches, the identification 
of the main components of the image is of great importance. For example, the automatic detection and 
tracking of moving objects in military applications and the separation of various products in industrial 
applications can be referred to as segmentation applications of the image. 

Detection is a fundamental stage for analyzing video images that are involved with the segmentation 
and separation of moving objects on the scene. This section not only focuses on the processing of the next 
steps but also significantly reduces the time required for computing. Subtraction from sequential frames, 
background subtraction, and optical flow are the most widely used methods for detecting. Due to 
environmental conditions such as camera movement, noise, lighting changes, shadows and repetitive 
movements of some parts of the background, such as branches of trees, the partitioning of moving objects 
is a relatively difficult problem, and its proper implementation is of considerable importance. In [17], 
they proposed a particular tracking algorithm for airplane based on R-FCN and Kalman Filter (KF). 
The great detection network, R-FCN, was used to provide the position message of a plane for 
the tracking model. 


2.1. Proposed tracking algorithm 

After discovering the purpose of the field and providing the appropriate presentation, it turns to 
track them. The purpose of tracking is to determine the position of objects in each image of the input 
sequence. To track objects, we need an algorithm to estimate the state of a dynamic system from incomplete 
and misleading measurements. 


2.2. Particle filter (PF) 

Nowadays, with the growth of methods including statistical methods, the problem of estimating 
a parameter or the state of a random nonlinear system along with observation and noise measurements 
is performed by the Monte Carlo estimation technique for solving complex statistical problems, 
which is expressed in the form of a particle filter algorithm [18]. As Kalman Filter (KF) is only limited to 
the Gaussian or normally distributed state variables and estimate them very poorly. This problem is overcome 
by the particle filter. Particle filter as used in [19, 20], can estimate the internal states from partial 
observations. This method is suitable for non-Gaussian and multi-model possibility density function. 
The particle filter is a new method for obtaining the probability distribution function based on the Bayesian 
theory. The particle filter algorithm is based on Monte Carlo(MC) chain methods in which a probability 
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density particle representation is used to estimate the parameters of an arbitrary distribution. The particle 
filter is the perfect solution for optimal purpose estimation in a situation where the nonlinear system model 
is non-Gaussian with the distribution of noise. The system's transition or movement model is expressed in 


terms of probability language, p */x, ): and the model of the observation or measurement of the system 


is expressed by p a )} X, representing the system state vector in the current Z, and representative 


View in the system. Considering the Bayesian recurrence equations that are made up of the following 
two steps [18]: 


Prediction Stage: 


p(“'/z,.,)= f P(X /X.1) PO /Z1) AX] (1) 
R? 


Update Stage: 


Xi Zt- 
C Xt) P(X/Z41) (2) 


P(Z;,/Z.1) 


pz) f P (=) p ) dx (3) 


The obtained models are usually non-linear and probabilistically possible to have non-Gaussian 
distributions and even multiples. One of the most well-known methods for solving the filtration problem 
is the EKF method. Since this method uses the Taylor Series estimator to use a nonlinear function, it has 
some error. One of the methods that can be used in such cases is estimated by particle filters. Integrating 
action takes place in the process of state-space variables, regardless of whether such an integral can be in any 
closed state, if the size of the space is high, it will have a high computational cost. The Monte Carlo 
simulation method, instead of calculating the integral overall points, performs its sampling at the points that 
have the largest share in the computation of the integral. The basis for this is to calculate the integral: 


I= { eax (4) 


Converts it as a product of the term g(x)=f(x)-p(x) where p(x) represents a probability function that 
satisfies the condition {f p(x)dx=1 and p(x)>0}, accordingly, with these conditions J is an Expected value of 
the optional function f(x). Now, if the model of the system state up to moment k, expressed by the function 
p(X, / Z,), is sampled independently of N's number of particles (N >> 1) and distribution such as (x9.,) [18]: 

s(§=1.2--.N)X,~ p(XilZp) (5) 


The estimated distribution of this distribution is defined as follows: 
Py (X,/Z,)& (6) 
N= 


Where 5(X,-Xk) represents the Dirac Delta function at a point (XX). Such an estimate is also 
used to distribute filtering. If the Expected value of the desired function f(0) is represented by I respect to 
P(X / Zy) : 


I= | Xk ) P(X / Zp dX = | f(Xy )- Pr(X / Zp dX, (7) 


Then, the Monte Carlo estimation of the integral as the sum of the values of the function f for 
the obtained samples is as follows: 


OIN 
=a 2, fxd» (8) 
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If samples Xj; are independent, then Iy is an unbiased estimate and can be approximated by 
the law of large numbers with I, and if N is a very large number, the estimate will converge to the real value. 
It is also shown that the convergence rate of estimation is independent of the variance and the dimensions of 
the characteristic space and depends only on the number of particles of N. Such a characteristic is the most 
important advantage of Monte Carlo integral to numerical integration methods [18]. In this paper, the particle 
filter algorithm predicts the position of the N particle in the range around the center of the mass of 
the previous frame randomly to begin to trace the particle. Each particle can be nominated for a new target 
position. By applying a kernel similar to the target model, the weighted histogram of the candidate particle 
model is obtained for each particle. The Bhattacharyya coefficient calculates the similarity between 
the object model and the object candidate particle model. The likelihood function detects particle visibility to 
enter the open sampling stage. The initial position of the moving object is estimated after the number of 
adaptive stages of the re-entry. 


3. SECOND PART: CLASSIFICATION TYPE OF FLYING OBJECTS (MILITARY OR 
NONMILITARY) 

It is worth remembering that in designing a system for classifying and categorizing an object, 
it is necessary to have a set of input data belonging to each desired category. From another point of view, 
it is possible to designate a system in such a way as to deal with an object belonging to one of the categories 
defined in terms of one or more specified categories and collecting the collection of their respective images, 
select it carefully to the category, and otherwise declare it to be an unknown or invalid object. The advantage 
of this method is that the system is reliable in categorizing valid objects that belong to one of the categories 
defined, logical coherence is observed among the members of each category, and more importantly, 
the description of the objectives is classified as desirable. In other words, according to the existing facilities 
and limitations, and the definition of several specific categories, specific and proportionate to the application, 
it is applied to system design. Since the further separation of objects into a classification, in addition to 
a more complete description of the target's purpose, is the ability and efficiency of the designed system. 
The selected categories of this paper are selected according to Figure 1. 


Aerial Objects 


Invalid Objects Valid Objects 
Nonmilitary Objects Military Objects 


Educational Aircraft Passenger Aircraft 


Hunting Aircraft Bomber Aircraft Transportation 
Aircraft 


Figure |. Categories of flying objects for designing this paper 
















Exploration 
Aircraft 


Therefore, in this paper, the two categories of valid objects and invalid objects are the desired 
outcomes of the system. In other words, the system is expected to be assigned to any flying object in contact 
with any of the defined categories, or to declare it categorized as inappropriate. Considering the above 
categories means that a collection of appropriate and comprehensive images as educational data of each 
category is available. 


3.1. Aircraft type detection system (military or nonmilitary) 

First, tracking flying objects that were done by the particle filter algorithm. Then, the tracked objects 
by the aircraft type detection system (military or nonmilitary aircraft), are investigated through four criteria 
including: size (center of mass) of objects, object speed vector, the diverse direction of motion of objects and 
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thermal imagery. Ultimately, the type of flying objects (military or nonmilitary aircraft) 
is determined. Four criteria for the aircraft type detection system are shown in Figure 2. 


Tracked Objects 


The size (center of 
mass) and the 
rectangular area of 
the objects 
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rectangle is small 


If the area of the 
rectangle is arge = 
Nonmilitary 
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Specific Aerial Path 






Path of the Aircraft 
Thermal Image 


Figure 2. Flowchart aircraft type detection system 


Uncertain paths = 
Military Aircraft 





Nonmilitary 
Aircraft 





3.2. Size (center of mass) of object 

First, it is needed to have binary images and objects are identified from the background. To obtain 
the center of the mass of the moving object, we need to sum up the quantities of x and y that are white in 
relation to the object and divide them into numbers [21]. In this way, the coordinates and position of 
the moving object are obtained. To track and identify changes, the moving object is detected and tracked by 
a rectangle with specified size or eight lines that radiate from the center of mass. 

In this criterion, initially, we need to convert the images to binary and then get the size and position 
of the tracked objects in successive frames. The method works in such a way that in the full screen the areas 
of the object are 1 and the background is 0. Then we compute the coordinates of the objects in the whole 
page (from top to bottom and from left to right) using equations (9) and (10) combine x together and y 
together, in which the mean values of x and y are obtained. Next, we get a total number of pixels. In the end 
to obtain the center of object mass using equation (11) we divide the average value of x and y into the total 
pixels. In general, the total size, wings, and center of mass of nonmilitary aircraft are larger than military 
aircraft. It is likely that according to the coordinates, it was decided that the object is military or nonmilitary. 
The maximum x is the tip of the plane. Determining the center of mass is for this point to be a fixed point 
relative to the points on the edge. An example of obtaining a center of mass is shown in Figure 3. 


X=) 9X (9) 

Y=" -0 a (10) 
X.Y 

center of mass= ———_——_ (11) 


total number of pixels 





(a) (b) 


Figure 3. An example of obtaining a center of mass, (a) Original image, (b) Detected planes 
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(c) 


Figure 3. An example of obtaining a center of mass, (c) Coordinate and center of mass (Continue) 


Figure 3 illustrates that there is a military aircraft and a nonmilitary aircraft in the frame, by 
obtaining the center of mass and the area of the aircraft rectangle, it can be concluded that the larger 
rectangular area of the object, is nonmilitary aircraft. Additionally, the rectangular area of the object, which is 
smaller, determines the military aircraft. 


3.3. Object speed vector 

At this stage, firstly, we need to measure the speed of the tracked objects in successive frames. 
Aircraft have different speeds. Nonmilitary aircraft such as regular passenger aircraft usually move at 
a steady pace, but military aircraft can travel at different speeds, the speed of these aircraft is much higher 
than the speed of nonmilitary aircraft. This feature can be used to detect a military aircraft from a nonmilitary 
when the object velocity decreases or increases over time, the object is the military aircraft, and if the object 
velocity is low and constant, the object is the nonmilitary aircraft. But the disadvantage of this method is that 
military aircraft can match their speed at the speed of nonmilitary aircraft, and it deceives the aircraft 
detection system. To calculate the object speed vector, first, we obtain the object position (x.y) in two 
different frames. Then, using equation (12), we obtain the Euclidean distance d between two different frames 
and, the object speed is obtained by inserting in the equation (13). (x, y) and t, are the position of 
the center of the object mass in the first frame and the time frame of the first frame, respectively. (x.y,) and t, 
are respectively, the position of the object mass center in the second frame and the time of the second frame. 


| 2 2 
Euclidean Distance=d= ,/(X,-X,) + (Y,-Y1) (12) 


Object Speed=V= — = (13) 
2-*1 


S 


3.4. Aircraft move path 

The main difference between military and nonmilitary aircraft is that nonmilitary aircraft travel on 
a specific path, while military aircraft move in different directions. By using this criterion, we can acquire 
flight paths. By obtaining a center of mass of objects in different frames, different points are obtained in each 
frame and by connecting these points, the direction of motion of the objects (aircraft) in the video, will be 
distinguished which can show the direction of movement of aircraft on a chart. Then it can be decided that 
aircraft that travel on a specific route are nonmilitary or military. The disadvantage of this criterion is that 
military aircraft can move in a certain direction and it makes the aircraft detection system deceived. 
Flight paths are one of the most important characteristics that can be considered for a flying object. 


3.5. Thermal imagery 

The range of thermal images is provided by imaging systems that have detectors that are sensitive to 
thermal radiation in the infrared electromagnetic spectrum. In general, imaging systems are a subset of 
electro-optic systems. Since an infrared heating system is responsible for preparing the input images of 
the system, familiarity with its fundamental concepts and structure, in general, can be effective in choosing 
the appropriate methods. The simulation of infrared thermal imagery is not possible as a solution to 
the problem of the rarity of heat-fitting images appropriate to the application, without being fully familiar 
with the structure of infrared imaging systems [22]. 

Thermal cameras and imaging have periodically been in interest mainly for smart environments, 
military applications. The increasing amount of image quality and resolution since the last few years have, 
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however, unlock new challenges and application areas in thermal imagery. Thermal cameras are useful to 
measure a temperature difference. These cameras are helpful compared to cameras operating in the visual 
spectrum due to their ability to acquire images in the complete night, robust to lighting variations and less 
interruption on privacy, as tempering is tedious [23]. The detection and tracking of an airborne infrared target 
in a complex struggle situation remain a challenging research area in infrared-imaging guidance [24, 25]. 
This criterion is not effective because the camera used in this article is not a thermal camera. 


4. SYSTEM SIMULATION TOOL 

To implement the detection sector, tracking and object estimation are required for a suitable digital 
processor. Furthermore, virtual simulation of the scene and imaging system will be possible using 
programmable digital processors. In this paper, a fixed camera is used for imaging, and the videos we want to 
process on them are virtualized with fixed background and with a computer. Personal computers are one of 
the best environments for applications like this. The advantage of using a computer in the simulation of 
the system is that the implementation of the detection and tracking section along with the estimated part 
using the main processor of the computer and its memory is well suited. Moreover, rapid access to lateral 
information and multimedia of the computer will facilitate the processes of comparison, evaluation and 
information transfer with greater speed and accuracy. The characteristics of the computer that is selected as 
the virtual simulation laboratory of the system are according to Table 1. The processor's programming and 
execution of the computational methods are based on the environment and facilities available in MATLAB 
software. The use of MATLAB software for its appropriate toolbox for processing digital data and image 
processing, as well as the ease of file management in it, has been considered as a virtual environment for 
simulating the system. 


Table 1. Computer system specifications used as a virtual laboratory 


Components Specifications 
Operating System Microsoft Windows 10 Enterprise 
RAM 4.00GB (3.89GB usable) 

CPU Intel(R) Core(TM) i5-4210U 
CPU@1.70GHz 2.40GHz 

Frame Grabber ASUS X552L Graphic Card 
Intel(R) HD NVIDIA 
GeForce820M 

Software MATLAB R2013a 


5. EXPERIMENTAL RESULTS 

To demonstrate the efficiency and resilience of the algorithm and the above system, several 
scenarios in various videos including challenges such as the number of objects (aircraft), the different paths, 
the direction of motion, the different speeds and various objects, have implemented. It is necessary to explain 
that in the forms of aircraft images, they are used for detecting and tracking the object. The yellow rectangle 
is pulled around the tracked objects, the red mark (*) indicates the real center of the object, which is extracted 
from the center of mass by the modeling method and the criterion of the center of mass, and the green star 
sign (*) represents the center of mass of an estimated object by particle filter as shown in Figure 4. 
The letters (Al, A2, ..) in the graphs indicate the direction of the movement of planes numbers(1,2.,...). 





(a) (b) 


Figure 4. Calculates the center of mass in different frames, (a) Frame 58, (b) Frame 110 
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Scenario: in the video, there are two passenger aircraft of different sizes, different directions, 
and two different military aircraft moving in different directions. Furthermore, the string of images has 219 
image frames with dimensions of 480x850. Consequently, the objects in this sequence of images are constant 
in terms of reshaping and changing size. First, aircraft that are moving are detected and they are separated 
from the background, the objects are white and the background is black as shown in Figure 5. Figure 6 
reveals that the particle filter algorithm could reliably track targets up to the end of the image string. The 
number of tracked objects is displayed as numbers 1, 2, and 3 above the targets. By calculating the center of 
mass and the area of the planet's rectangles: The area of aircraft numbers 2 and 3 is more than the aircraft 1 
and 4. According to the results of the aircraft speeds in Table 2, aircrafts numbers | and 4 have a higher 
speed than aircrafts numbers 2 and 3. Figure 7 shows that the direction of the movement of aircrafts numbers 
1, 2 and 3 are in a straight line, and the direction of aircraft number 4 is not a straight line. Hence, according 
to the results, we conclude that the aircraft number | and 4 are military aircraft and aircraft number 2 and 3 
nonmilitary aircraft. 


Table 2. Aircraft speed results 


Speed Achieve F Aircraft Specifications 


i 






207 





Number 1 


ae 
120 es ré 


Number 2 


119 





Number 3 


368 





Number 4 





(a) (b) 


Figure 5. Objects detection in different frames, (a) Frame 58, (b) Frame 110 
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(a) (b) 
Figure 6. Objects tracking by particle filter in different frames, (a) Frame 58, (b) Frame 110 
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Figure 7. Aircraft movement paths 


6. CONCLUSION 

In conclusion, this paper contains two main parts, the first part is the detection and tracking of flying 
objects. At first, the aircraft that are moving are detected and separated from the background, the object 
is white and the background is black. Flying objects are tracked using the particle filter algorithm. 
The particle filter algorithm predicts the position of N particle in the range around the center of mass 
of the previous frame randomly to start tracking. Each particle can be nominated for a new target position. 
The second part is to classify flying objects (military or nonmilitary), based on four criteria; Size (center 
of mass) of objects, object speed vector, the direction of motion of objects, and thermal imagery identifies 
the type of tracked flying objects. Detection and tracking based on the proposed method and aircraft type 
detection system have been implemented on a large number of images. To demonstrate the efficiency 
and resilience of the algorithm and the above system, several scenarios in various videos have been tested 
that include challenges such as the number of objects (aircraft), the different paths, the divers’ directions of 
motion, different speeds and various objects. In this paper, only the results of one scenario are mentioned. 
According to the results of the implementation of the proposed system, detection, tracking and aircraft type 
detection system for a variety of different images has been adequately answered. 
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